1) I think we're supposed to assume more than 500 zombies surrounding the Sanctuary -- more like 5-10k? Dunno. Even though some scenes only show a couple hundred zombies, we can blame that on a) budget limitations, b) bad job of staging the scene (which happened a few times during this season)

2) Yes, they could've shot all the zombies, but Negan wanted to conserve ammo. Of course, the show has been wishy-washy with the ammo excuse -- one day, our heroes need to conserve ammo. The next day, they're shooting like a 1980s action movie.

3) Also making it more difficult to eliminate all the zombies -- Rick's crew had snipers, and they were covering all the windows & doors (supposedly)

I don't think Rick's plan was great to begin with, and then it wasn't explained or "shown" very well on screen -- but I think the intent was only to delay & annoy Negan for 48-72 hours, not necessarily keep him trapped forever.

When there was no immediate danger, Negan was content to stay put (and plan his next steps). But when the walls were busted, that forced Negan to escape & counter-attack.